{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can download the `requirements.txt` for this course from the workspace of this lab. `File --> Open...`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r8teiBZ6dP5g"
   },
   "source": [
    "# Lesson 2: Data Types and Sizes\n",
    "\n",
    "In this lab, you will learn about the common data types used to store the parameters of machine learning models.\n",
    "\n",
    "\n",
    "The libraries are already installed in the classroom.  If you're running this notebook on your own machine, you can install the following:\n",
    "\n",
    "```Python\n",
    "!pip install torch==2.1.1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29,
    "id": "x_9J8WavkQGl"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29,
    "id": "nFC9wgzxBZvz"
   },
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hW7psx41rn0h"
   },
   "source": [
    "### Integers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# Information of `8-bit unsigned integer`\n",
    "torch.iinfo(torch.uint8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# Information of `8-bit (signed) integer`\n",
    "torch.iinfo(torch.int8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "### Information of `64-bit (signed) integer`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "### Information of `32-bit (signed) integer`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "### Information of `16-bit (signed) integer`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Floating Points "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# by default, python stores float data in fp64\n",
    "value = 1/3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": [
    "format(value, '.60f')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# 64-bit floating point\n",
    "tensor_fp64 = torch.tensor(value, dtype = torch.float64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": [
    "print(f\"fp64 tensor: {format(tensor_fp64.item(), '.60f')}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 63
   },
   "outputs": [],
   "source": [
    "tensor_fp32 = torch.tensor(value, dtype = torch.float32)\n",
    "tensor_fp16 = torch.tensor(value, dtype = torch.float16)\n",
    "tensor_bf16 = torch.tensor(value, dtype = torch.bfloat16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 80
   },
   "outputs": [],
   "source": [
    "print(f\"fp64 tensor: {format(tensor_fp64.item(), '.60f')}\")\n",
    "print(f\"fp32 tensor: {format(tensor_fp32.item(), '.60f')}\")\n",
    "print(f\"fp16 tensor: {format(tensor_fp16.item(), '.60f')}\")\n",
    "print(f\"bf16 tensor: {format(tensor_bf16.item(), '.60f')}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 294,
     "status": "ok",
     "timestamp": 1699482682456,
     "user": {
      "displayName": "Marc Sun",
      "userId": "00829270524676809963"
     },
     "user_tz": 300
    },
    "height": 46,
    "id": "hUukczHrBodt",
    "outputId": "4e75bdf1-25b6-4e0c-9c08-18208e943b10"
   },
   "outputs": [],
   "source": [
    "# Information of `16-bit brain floating point`\n",
    "torch.finfo(torch.bfloat16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# Information of `32-bit floating point`\n",
    "torch.finfo(torch.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "### Information of `16-bit floating point`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "### Information of `64-bit floating point`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3DbtWVVc_jiW"
   },
   "source": [
    "### Downcasting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46,
    "id": "JHDsqfauBID9"
   },
   "outputs": [],
   "source": [
    "# random pytorch tensor: float32, size=1000\n",
    "tensor_fp32 = torch.rand(1000, dtype = torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** As it is random, the values you get will be different from the video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# first 5 elements of the random tensor\n",
    "tensor_fp32[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46,
    "id": "xXLANbSx_nx4"
   },
   "outputs": [],
   "source": [
    "# downcast the tensor to bfloat16 using the \"to\" method\n",
    "tensor_fp32_to_bf16 = tensor_fp32.to(dtype = torch.bfloat16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": [
    "tensor_fp32_to_bf16[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# tensor_fp32 x tensor_fp32\n",
    "m_float32 = torch.dot(tensor_fp32, tensor_fp32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": [
    "m_float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 46
   },
   "outputs": [],
   "source": [
    "# tensor_fp32_to_bf16 x tensor_fp32_to_bf16\n",
    "m_bfloat16 = torch.dot(tensor_fp32_to_bf16, tensor_fp32_to_bf16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": [
    "m_bfloat16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note\n",
    "- You'll use \"downcasting\" as a simple form of quantization in the next lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "height": 29
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "authorship_tag": "ABX9TyNKuOYgD21Zty8Z1Smilnjc",
   "collapsed_sections": [
    "x_9J8WavkQGl",
    "sulufN1wkK_L",
    "hqPUM0f8oGgf",
    "TgmPMm-ZvdXX",
    "MJuh8aqo9LO-",
    "3bOevU0Ez4KB",
    "LgtYIhSf0Uu0",
    "9DH7-tDDvK5N",
    "TuZhoPK6weuR",
    "WAOeSNraxZeF",
    "UGXo-IljxmNG",
    "_N3G-dX32awT",
    "hW7psx41rn0h",
    "OmcNPw6zlRp7",
    "3_zvORGrnTR8",
    "jh9IuiovrnoF",
    "isDcbkXxxiTf",
    "wM-xkASw1odi",
    "3DbtWVVc_jiW",
    "ViImpV5rAvyp",
    "RdcyknnjBD99",
    "-eYj4UUXCAlJ",
    "MCLe1N4GCSQT",
    "GIY7IrOv_3cD",
    "8HlFKDBKGNG8"
   ],
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
